| As the repeated testing of unchanged functionality (regression testing) is the most sizeable and time-consuming part of the
test, tools for automated test execution are attractive to many organisations. Regression testing starts as early as when a
system is being built and takes up an increasing part of test work during the life cycle of the system (see
figure 1 - Increasing share of regression testing during the lifecycle). The automated execution of such
regression tests can save time. This is attractive not only to the tester, who is relieved of repetitive and therefore
boring daily activities, but also to the calculating test manager who can save tens of percents.
Figure 1: Increasing share of regression testing during the lifecycle
There are two variants of this test tool type:
-
Tools that automate test execution via the user interface (GUI) of the application to be tested. These are also
called record & playback tools. A record & playback tool records the test input (data and actions) and the
expected result in a script. The tool can play back the script at a later time, so that the test can be repeated
easily (please note that the term ‘script’ in this context should not be confused with the manual test scripts that
are part of the test specifications).
-
Tools that automate test execution via a program interface. Examples of a program interface are Application
Programming Interface (API) or messages in XML format. Often this tool type offers the possibility of mutating
stored input data and provides support when generating test input. Generally speaking, these tools are combined
with comparison tools to enable analysis of the test results.
The great advantage of automated test execution tools is that a test can be repeated by automation at a later stage.
This advantage is nullified if the test object is changed in such a way that the automated script blocks during
playback. Maintenance to the automated scripts is necessary to use the tool efficiently. Such maintenance should not
cost more than the benefit yielded by automated test execution. Changes in the test object must result in a limited
number of changes in the automated scripts. This is often the case in regression testing, so that this tool type is
extremely suitable for this test type.
The combination of tool, framework, test cases, automated test scripts, and recorded results is called a test suite.
The framework in a test suite is a library of reusable automated scripts. Each script is in fact a small program. Use
of the basic principles of modular programming increases the maintainability of the scripts: each group of successive
actions that must be carried out repeatedly (for example moving to a certain screen in the application) is best stored
as a separate module. If something changes in the group of activities (for example because of a different menu setup),
then only one module will need to be adapted. Modules exist at different levels of abstraction, varying from activating
or checking a specific object of the system to be tested, to carrying out a business process. Having such an
architecture makes it possible for new test suites (for new systems) to be created in a short period of time, because
many of the necessary building blocks (modules) are already present in the library. To construct a test suite in such a
modular fashion, expertise in the fields of testing and software development is required. The required effort to adapt
a test suite (and therefore also the framework) for a new release must not outweigh the benefits of the use of the test
suite. The main quality requirements for a test suite are: maintainable, flexible, robust and reusable (see also
[Fewster, 1991]).
|
